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Abstract 

The first two years of user service of the third generation 
hght source BESSY II emphasized the importance of a re- 
liable, comprehensive and dense logging of a few thousand 
setpoints, readbacks, status and alarm values. Today data 
from sources with various characteristics residing in differ- 
ent protected networks are centrally collected and retriev- 
able via an uncomplex CGI program to any desktop sys- 
tem on the site. Data post-processing tools cover Windows 
applications, IDL, SDDS and custom programs matching 
users skills and preferences. In this paper illustrative sam- 
ple data explorations are described that underline the im- 
portance of the logging system for operations as well as 
for the understanding of singular events or long term drifts. 
Serious shortcomings of the present installation and focus 
of further development are described. 

1 INTRODUCTION 

Like other third generation light sources BESSY II exceeds 
many of the primary design goals. Users not only appreci- 
ate the additional potential of the excellent beam definition 
and stability — an increasing number of experiments sim- 
ply depend on the high and reliable beam quality. Espe- 
cially important are minimal beam center of mass drifts, 
well defined beam energy with minimal spread and high 
beam intensity with a long lifetime. It is difficult to prevent 
drifting of these parameters over the several days necessary 
for an experiment. Many effects originating from facil- 
ity operating conditions and user activities can contribute. 
Since there is never enough time to isolate all possible ef- 
fects by dedicated accelerator development studies archives 
of logged data are most important sources of information. 

2 SETUP AND STATUS 

Despite the eminent importance of archived data the archiv- 
ing system at BESSY is far from being well settled. Ade- 
quate carefully done is the collection both of snapshot files 
and long term monitoring data [|lj|. Loss or omission of es- 
sential data would destroy unrecoverable knowledge about 
past behaviour of the facility. Retrieval tools are still cum- 
bersome, immature and subject of maloperation frequently 
resulting in loss of work time. Configuration is mainly 
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hand-work, thus not fault free. Surveillance of data source 
availability and data integrity is done occasionally. Only 
the collector programs themselves are systematically su- 
pervised by watch-dog or stop/restart procedures. 

2. 1 SDDS based Data Store 

Initially the BESSY archiving configuration was based on 
the SDDS toolkit. Storage format are compressed SDDS 
files spanning a device class and a full day, sorted into a 
calendar mapping directory structure. A TclTk glue appli- 
cation combines navigation, SDDS data retrieval, correla- 
tion and export [|l]] . 

This data store is still a good compromise even though 
not optimal with respect to data format and size, network 
resources and CPU requirements: channel selection, pre- 
viewing facility, available post-processing tools cover most 
of operators requirements. The SDDS archive is not dis- 
continued, collects 20 GB/y and serves as valuable backup 
system. A more or less frozen and easy maintainable list of 
signals essential for the understanding of basic operation 
parameters are monitored. Major obstacle for a site-wide 
usage of the archive is the (intended) in-accessibility of the 
data store residing in the protected accelerator control pro- 
duction area. 

2.2 Central Channel Archiver 

Since mid 2000 a Channel Archiver |Q instance has been 
set up in addition. It is intended to overcome the self- 
containment of the (accelerator) SDDS archiver and serve 
the whole site. Any major development and configura- 
tion effort goes into this system. Data collector engine(s) 
and CGIExport retrieval tools are installed in a dedi- 
cated environment A six processor HP N-class server 
{archive server) in a non-routable private network stores 
the data on a RAID system that is backed up to a tape robot. 
It is planned to migrate mass storage to a fibre channel sys- 
tem attached to a tape library this year. 

2.3 Data Flow 

In an attempt to minimize adverse effects on the system 
caused by unexpected activities and to maximize uptime 
neither user accounts nor NFS access to the archiver net- 
work are provided. For data collection all data sources re- 
siding on dedicated networks are connected by two multi- 
homed CA-gateway computers (8 network interfaces each). 



Presently a single archiving engine (process) stores 50 
GB/y accelerator relevant data. A second engine has been 
set up early this year for the beamline area and auxiliary 
data presently collecting about 15GB/y. 

Common retrieval method is HTTP invocation of 
CGIExport [|[| via the central network router Typically 
the available gnuplot presentation of the data requested 
is used as a preview ensuring that the data selection pro- 
vides the desired information. Then the data are retrieved 
in spread-sheet or matlab format and stored on local disk. 
Favourite postprocessing tools are PC Windows tools (Ori- 
gin, Excel) or UNIX applications (IDL, Matlab). A small 
program (caa2sdds) converts the spread-sheet output to 
SDDS format enabling data analysis with the full data se- 
lection, post-processing and display power of SDDS. 

3 TYPICAL UTILIZATION 

3. 1 Identification of Singular Events 

Probably tracking down sudden perturbations to its causes 
is the most common usage of the archive. Examples for this 
application are e.g. an unusual large drift that corresponded 
to the failure of a water pump or the sudden onset of orbit 
jumps that was due to an improper motor reset resulting in 
a constant rotation of strong chicane magnets. 

3.2 First Hints on Unexpected Effects 

Archived data help to get a first idea of possible explana- 
tions: Mid 2001 for example a strong, periodic orbit pertur- 
bation has been reported by the operators. By phase anal- 
ysis it was possible to locate the problem source with a 
few meters precision at a ring segment where no active ele- 
ments are installed. The time pattern of perturbation onset 
and disappearance (see fig. suggested an unknown cor- 
relation with user activities. Targeted investigation found 
out that one user group reversed the field of a 1 [T] magnet 
twice a minute several meters apart from the beampipe. 



3.3 Analysis of Changes 




Figure 1: Orbit perturbations due to switched user mag- 
net outside the storage ring tunnel. Phases of experimental 
activity are clearly visible. Viewing tool: Xarr. 
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Figure 2: Vacuum effects on lifetime shown as a function 
of accumulated beam dose. Postprocessing tool: EXCEL 

On the long term extreme the archive provides the data 
needed to make evolutions visible that are hardly percep- 
tible on a fill to fill basis. Plotting e.g. the normalized 
lifetime [mAh] against the accumulated dose [Ah] over the 
full operating time of the facility is a powerful mean to find 
out very fundamental factors: From fig. ^ it can be con- 
cluded, that the vacuum related lifetime reduction is basi- 
cally overcome by beam scrubbing 1000 [Ah] after start 
up of the accelerator Every venting due to installation re- 
quirements needs another 100 [Ah] to reinstall the previous 
performance. On top of these basic conditions global life- 
time improving effects of Landau cavities (mid 2000) as 
well as reducing effects of imperfectly corrected insertion 
devices (beginning 2001) can be seen. 

4 DEMANDING REQUIREMENTS 

4.1 Uptime, Reliability 

Requirements on uptime, reliability and consistency of the 
archive are substantial. The archive data have to contain 
signals of very different importance. Beam intensity e.g. is 
analyzed and correlated in any thinkable way: integration 
(dose), differentiation (beam loss), pattern analysis (user 
runs) etc. Here a loss of data would be serious, but rec- 
ognized within minutes. Other signals are monitored as a 
precaution. They could potentially help to find candidates 
for sources of performance degradation. Dispensable for 
the all day business they are not under human surveillance. 
Regardless they have to contain reliable data when needed. 



4.2 Data Density, Aging 

The most common approaches to prevent growing of the 
archive to unmanageable dimensions are removal of 'old' 
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Figure 3: Raw data of (uncorrected) vertical orbit stability 
(+/- 5 RMS) during all user fills (220 mA - 80 mA) at 
user run #4 (Aug. 2000, left). General degradation and spu- 
rious exotic drifts can be clearly identified at run #5 (Oct. 
2000, right). Postprocessing Tool: Origen 

data (tape, deletion) or a progressive reduction of data den- 
sity. Fig. H and ^ are examples of the opposite require- 
ment for a dense and long term archive. In Fig. ^ spurious 
observations and user complaints could be quantified after 
serious hardware modifications. The comparison of perfor- 
mance and influence of a new operation mode required per 
fill details (8h) months apart for fig. ^ 

5 PRESENT FOCUS OF ACTIVITIES 

5. 1 Data Collector 

Today management and configuration of collector engines 
is further robustified. Usage of the system is simplified 
by GUI administration tools. Signal configuration manage- 
ment based on the reference RDB is still missing. 

5.2 Retrieval 

Performance of data retrieval from large and multiple 
archives has been drastically enhanced. Channel detection 
method for a given time interval is improved. Volume of 
intermediate data needed for previewing is reduced to the 
minimum allowed by the anticipated gnuplot resolution. 

5.3 Data Partitioning 

From the iterator model and the hash table directories the 
binary data format of the Channel Archiver is optimized for 
retrieval of data from archives containing a moderate num- 
ber of channels and starting e.g. from 'now' going back- 
wards in time. Retrieving a dozen of channels out of the 
'middle' of a continuous archive holding several thousands 
of channels requires patience. 

As a first improvement approach the huge monolithic 
data block is split into a moderate number of weekly or- 
dered chunks holding certain fragments of the whole signal 
collection. Adjustment of the I/O routines results in orders 
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Figure 4: Stability comparison of 'Uncorrected' with 'Drift 
Corrected' Fills. Data Postprocessing Tool: IDL. 

of magnitude retrieval acceleration. But however home 
grown data formats are optimized: ultimately the retrieval 
of arbitrary data selections out of huge data stores is best 
done with commercial RDB systems. Consequently the uti- 
lization of a RDB storage format has to be re-considered. 

6 SUMMARY 

Ideally one would like to be able to 'replay' any control- 
lable and measurable parameter out of the signal archive 
with the reasonable time resolution of a few seconds. For a 
BESSY size facility this would require data stores of sev- 
eral TB/y. The Channel Archiver provides a robust data 
collector and retrieval toolkit but the archive itself has to 
be reduced to manageable dimensions. 

The challenges today are configuration (select relevant 
signals, grouping, choose proper archiving frequencies), 
correlation detection (identify signals) and data organisa- 
tion (optimized search). Plotting options and postprocess- 
ing requirements have to be provided by the end-user ac- 
cording to his specific skills and varying needs. 
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